{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "---\n",
    "description: Design Your Continual Learning Experiments\n",
    "---\n",
    "# Putting All Together\n",
    "\n",
    "Welcome to the \"_Putting All Together_\" tutorial of the \"_From Zero to Hero_\" series. In this part we will summarize the major Avalanche features and how you can put them together for your continual learning experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "!pip install avalanche-lib==0.5"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 🛴 A Comprehensive Example\n",
    "\n",
    "Here we report a complete example of the _Avalanche_ usage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting experiment...\n",
      "Start of experience:  0\n",
      "Current Classes:  [3, 4]\n",
      "-- >> Start of training phase << --\n",
      "100%|██████████| 24/24 [00:01<00:00, 12.81it/s]\n",
      "Epoch 0 ended.\n",
      "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 549773.7910\n",
      "\tDiskUsage_MB/train_phase/train_stream/Task000 = 549773.7910\n",
      "\tLoss_Epoch/train_phase/train_stream/Task000 = 1.1541\n",
      "\tLoss_MB/train_phase/train_stream/Task000 = 0.3121\n",
      "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0015\n",
      "\tTime_Epoch/train_phase/train_stream/Task000 = 1.8238\n",
      "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.7195\n",
      "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.9577\n",
      "-- >> End of training phase << --\n",
      "Training completed\n",
      "Computing accuracy on the whole test set\n",
      "-- >> Start of eval phase << --\n",
      "-- Starting eval on experience 0 (Task 0) from test stream --\n",
      "100%|██████████| 20/20 [00:01<00:00, 17.38it/s]\n",
      "> Eval on experience 0 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 102.7259\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 549774.4395\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 0.2281\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.9844\n",
      "-- Starting eval on experience 1 (Task 0) from test stream --\n",
      "100%|██████████| 19/19 [00:01<00:00, 17.59it/s]\n",
      "> Eval on experience 1 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 102.7046\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 549774.7803\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 4.9910\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.0000\n",
      "-- Starting eval on experience 2 (Task 0) from test stream --\n",
      "100%|██████████| 21/21 [00:01<00:00, 17.99it/s]\n",
      "> Eval on experience 2 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 102.5469\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 549775.1211\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 5.4075\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.0000\n",
      "-- Starting eval on experience 3 (Task 0) from test stream --\n",
      "100%|██████████| 22/22 [00:01<00:00, 17.61it/s]\n",
      "> Eval on experience 3 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 102.6420\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 549775.4619\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 4.5129\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.0000\n",
      "-- Starting eval on experience 4 (Task 0) from test stream --\n",
      "100%|██████████| 21/21 [00:01<00:00, 17.87it/s]\n",
      "> Eval on experience 4 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 102.6191\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 549775.8027\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 4.8327\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n",
      "-- >> End of eval phase << --\n",
      "\tConfusionMatrix_Stream/eval_phase/test_stream = \n",
      "tensor([[   0,    0,    0,  768,  212,    0,    0,    0,    0,    0],\n",
      "        [   0,    0,    0, 1048,   87,    0,    0,    0,    0,    0],\n",
      "        [   0,    0,    0,  696,  336,    0,    0,    0,    0,    0],\n",
      "        [   0,    0,    0,  990,   20,    0,    0,    0,    0,    0],\n",
      "        [   0,    0,    0,   11,  971,    0,    0,    0,    0,    0],\n",
      "        [   0,    0,    0,  614,  278,    0,    0,    0,    0,    0],\n",
      "        [   0,    0,    0,  131,  827,    0,    0,    0,    0,    0],\n",
      "        [   0,    0,    0,  283,  745,    0,    0,    0,    0,    0],\n",
      "        [   0,    0,    0,  669,  305,    0,    0,    0,    0,    0],\n",
      "        [   0,    0,    0,  110,  899,    0,    0,    0,    0,    0]])\n",
      "\tDiskUsage_Stream/eval_phase/test_stream/Task000 = 549775.8027\n",
      "\tLoss_Stream/eval_phase/test_stream/Task000 = 3.9915\n",
      "\tStreamForgetting/eval_phase/test_stream = 0.0000\n",
      "\tTop1_Acc_Stream/eval_phase/test_stream/Task000 = 0.1961\n",
      "Start of experience:  1\n",
      "Current Classes:  [5, 6]\n",
      "-- >> Start of training phase << --\n",
      "100%|██████████| 23/23 [00:01<00:00, 14.08it/s]\n",
      "Epoch 0 ended.\n",
      "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 549789.3984\n",
      "\tDiskUsage_MB/train_phase/train_stream/Task000 = 549789.3984\n",
      "\tLoss_Epoch/train_phase/train_stream/Task000 = 2.1353\n",
      "\tLoss_MB/train_phase/train_stream/Task000 = 0.4682\n",
      "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0009\n",
      "\tTime_Epoch/train_phase/train_stream/Task000 = 1.5832\n",
      "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.4197\n",
      "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.9027\n",
      "-- >> End of training phase << --\n",
      "Training completed\n",
      "Computing accuracy on the whole test set\n",
      "-- >> Start of eval phase << --\n",
      "-- Starting eval on experience 0 (Task 0) from test stream --\n",
      "100%|██████████| 20/20 [00:01<00:00, 17.60it/s]\n",
      "> Eval on experience 0 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 102.7213\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 549790.0469\n",
      "\tExperienceForgetting/eval_phase/test_stream/Task000/Exp000 = 0.9844\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.7269\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0000\n",
      "-- Starting eval on experience 1 (Task 0) from test stream --\n",
      "100%|██████████| 19/19 [00:01<00:00, 17.63it/s]\n",
      "> Eval on experience 1 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 102.8530\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 549790.4814\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 0.3979\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.9238\n",
      "-- Starting eval on experience 2 (Task 0) from test stream --\n",
      "100%|██████████| 21/21 [00:01<00:00, 17.95it/s]\n",
      "> Eval on experience 2 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 102.8862\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 549790.8223\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 5.2694\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.0000\n",
      "-- Starting eval on experience 3 (Task 0) from test stream --\n",
      "100%|██████████| 22/22 [00:01<00:00, 18.21it/s]\n",
      "> Eval on experience 3 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 103.0279\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 549791.1631\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 4.2881\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.0000\n",
      "-- Starting eval on experience 4 (Task 0) from test stream --\n",
      "100%|██████████| 21/21 [00:01<00:00, 17.82it/s]\n",
      "> Eval on experience 4 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 102.6138\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 549791.5039\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 4.7604\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n",
      "-- >> End of eval phase << --\n",
      "\tConfusionMatrix_Stream/eval_phase/test_stream = \n",
      "tensor([[  0,   0,   0,   0,   0, 666, 314,   0,   0,   0],\n",
      "        [  0,   0,   0,   0,   0, 912, 223,   0,   0,   0],\n",
      "        [  0,   0,   0,   0,   0, 212, 820,   0,   0,   0],\n",
      "        [  0,   0,   0,   0,   0, 904, 106,   0,   0,   0],\n",
      "        [  0,   0,   0,   0,   0, 128, 854,   0,   0,   0],\n",
      "        [  0,   0,   0,   0,   0, 770, 122,   0,   0,   0],\n",
      "        [  0,   0,   0,   0,   0,  19, 939,   0,   0,   0],\n",
      "        [  0,   0,   0,   0,   0, 743, 285,   0,   0,   0],\n",
      "        [  0,   0,   0,   0,   0, 754, 220,   0,   0,   0],\n",
      "        [  0,   0,   0,   0,   0, 428, 581,   0,   0,   0]])\n",
      "\tDiskUsage_Stream/eval_phase/test_stream/Task000 = 549791.5039\n",
      "\tLoss_Stream/eval_phase/test_stream/Task000 = 3.5491\n",
      "\tStreamForgetting/eval_phase/test_stream = 0.9844\n",
      "\tTop1_Acc_Stream/eval_phase/test_stream/Task000 = 0.1709\n",
      "Start of experience:  2\n",
      "Current Classes:  [8, 2]\n",
      "-- >> Start of training phase << --\n",
      "100%|██████████| 24/24 [00:01<00:00, 14.30it/s]\n",
      "Epoch 0 ended.\n",
      "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 549805.4072\n",
      "\tDiskUsage_MB/train_phase/train_stream/Task000 = 549805.4072\n",
      "\tLoss_Epoch/train_phase/train_stream/Task000 = 2.4022\n",
      "\tLoss_MB/train_phase/train_stream/Task000 = 0.6468\n",
      "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0008\n",
      "\tTime_Epoch/train_phase/train_stream/Task000 = 1.6303\n",
      "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.3785\n",
      "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.9094\n",
      "-- >> End of training phase << --\n",
      "Training completed\n",
      "Computing accuracy on the whole test set\n",
      "-- >> Start of eval phase << --\n",
      "-- Starting eval on experience 0 (Task 0) from test stream --\n",
      "100%|██████████| 20/20 [00:01<00:00, 18.51it/s]\n",
      "> Eval on experience 0 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 102.9407\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 549806.0557\n",
      "\tExperienceForgetting/eval_phase/test_stream/Task000/Exp000 = 0.9844\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.8448\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0000\n",
      "-- Starting eval on experience 1 (Task 0) from test stream --\n",
      "100%|██████████| 19/19 [00:01<00:00, 18.07it/s]\n",
      "> Eval on experience 1 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 102.9247\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 549806.4902\n",
      "\tExperienceForgetting/eval_phase/test_stream/Task000/Exp001 = 0.9238\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 2.1683\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.0000\n",
      "-- Starting eval on experience 2 (Task 0) from test stream --\n",
      "100%|██████████| 21/21 [00:01<00:00, 17.77it/s]\n",
      "> Eval on experience 2 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 102.6346\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 549814.9121\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 0.5857\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.9412\n",
      "-- Starting eval on experience 3 (Task 0) from test stream --\n",
      "100%|██████████| 22/22 [00:01<00:00, 18.27it/s]\n",
      "> Eval on experience 3 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 102.8152\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 549815.2529\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 3.6369\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.0000\n",
      "-- Starting eval on experience 4 (Task 0) from test stream --\n",
      "100%|██████████| 21/21 [00:01<00:00, 17.97it/s]\n",
      "> Eval on experience 4 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 102.7778\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 549815.5938\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 3.8625\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n",
      "-- >> End of eval phase << --\n",
      "\tConfusionMatrix_Stream/eval_phase/test_stream = \n",
      "tensor([[  0,   0, 413,   0,   0,   0,   0,   0, 567,   0],\n",
      "        [  0,   0, 492,   0,   0,   0,   0,   0, 643,   0],\n",
      "        [  0,   0, 973,   0,   0,   0,   0,   0,  59,   0],\n",
      "        [  0,   0, 325,   0,   0,   0,   0,   0, 685,   0],\n",
      "        [  0,   0, 222,   0,   0,   0,   0,   0, 760,   0],\n",
      "        [  0,   0,  83,   0,   0,   0,   0,   0, 809,   0],\n",
      "        [  0,   0, 775,   0,   0,   0,   0,   0, 183,   0],\n",
      "        [  0,   0,  99,   0,   0,   0,   0,   0, 929,   0],\n",
      "        [  0,   0,  59,   0,   0,   0,   0,   0, 915,   0],\n",
      "        [  0,   0,  84,   0,   0,   0,   0,   0, 925,   0]])\n",
      "\tDiskUsage_Stream/eval_phase/test_stream/Task000 = 549815.5938\n",
      "\tLoss_Stream/eval_phase/test_stream/Task000 = 2.6406\n",
      "\tStreamForgetting/eval_phase/test_stream = 0.9541\n",
      "\tTop1_Acc_Stream/eval_phase/test_stream/Task000 = 0.1888\n",
      "Start of experience:  3\n",
      "Current Classes:  [9, 1]\n",
      "-- >> Start of training phase << --\n",
      "100%|██████████| 26/26 [00:01<00:00, 14.39it/s]\n",
      "Epoch 0 ended.\n",
      "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 549830.1592\n",
      "\tDiskUsage_MB/train_phase/train_stream/Task000 = 549830.1592\n",
      "\tLoss_Epoch/train_phase/train_stream/Task000 = 2.1396\n",
      "\tLoss_MB/train_phase/train_stream/Task000 = 0.7244\n",
      "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0007\n",
      "\tTime_Epoch/train_phase/train_stream/Task000 = 1.7580\n",
      "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.3703\n",
      "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.9215\n",
      "-- >> End of training phase << --\n",
      "Training completed\n",
      "Computing accuracy on the whole test set\n",
      "-- >> Start of eval phase << --\n",
      "-- Starting eval on experience 0 (Task 0) from test stream --\n",
      "100%|██████████| 20/20 [00:01<00:00, 18.27it/s]\n",
      "> Eval on experience 0 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 102.8789\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 549830.8076\n",
      "\tExperienceForgetting/eval_phase/test_stream/Task000/Exp000 = 0.9844\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.7845\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0000\n",
      "-- Starting eval on experience 1 (Task 0) from test stream --\n",
      "100%|██████████| 19/19 [00:01<00:00, 18.62it/s]\n",
      "> Eval on experience 1 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 102.8579\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 549831.2422\n",
      "\tExperienceForgetting/eval_phase/test_stream/Task000/Exp001 = 0.9238\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 2.6320\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.0000\n",
      "-- Starting eval on experience 2 (Task 0) from test stream --\n",
      "100%|██████████| 21/21 [00:01<00:00, 17.98it/s]\n",
      "> Eval on experience 2 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 102.8979\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 549831.6768\n",
      "\tExperienceForgetting/eval_phase/test_stream/Task000/Exp002 = 0.6171\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 1.4709\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.3240\n",
      "-- Starting eval on experience 3 (Task 0) from test stream --\n",
      "100%|██████████| 22/22 [00:01<00:00, 17.56it/s]\n",
      "> Eval on experience 3 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 102.6780\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 549832.1113\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 0.5977\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.9725\n",
      "-- Starting eval on experience 4 (Task 0) from test stream --\n",
      "100%|██████████| 21/21 [00:01<00:00, 18.08it/s]\n",
      "> Eval on experience 4 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 102.7687\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 549832.4521\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 3.6372\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n",
      "-- >> End of eval phase << --\n",
      "\tConfusionMatrix_Stream/eval_phase/test_stream = \n",
      "tensor([[   0,   11,  354,    0,    0,    1,    0,    0,  110,  504],\n",
      "        [   0, 1132,    1,    0,    0,    0,    0,    0,    0,    2],\n",
      "        [   0,  317,  617,    0,    0,    0,    0,    0,    1,   97],\n",
      "        [   0,  621,  131,    0,    0,    0,    0,    0,   75,  183],\n",
      "        [   0,   69,    9,    0,    0,    0,    0,    0,    0,  904],\n",
      "        [   0,  212,   61,    0,    0,    0,    0,    0,  118,  501],\n",
      "        [   0,  155,  549,    0,    0,    0,    0,    0,    3,  251],\n",
      "        [   0,  174,    1,    0,    0,    0,    0,    0,    0,  853],\n",
      "        [   0,  400,   24,    0,    0,    0,    0,    0,   33,  517],\n",
      "        [   0,   50,    6,    0,    0,    0,    0,    0,    0,  953]])\n",
      "\tDiskUsage_Stream/eval_phase/test_stream/Task000 = 549832.4521\n",
      "\tLoss_Stream/eval_phase/test_stream/Task000 = 2.1951\n",
      "\tStreamForgetting/eval_phase/test_stream = 0.8418\n",
      "\tTop1_Acc_Stream/eval_phase/test_stream/Task000 = 0.2735\n",
      "Start of experience:  4\n",
      "Current Classes:  [0, 7]\n",
      "-- >> Start of training phase << --\n",
      "100%|██████████| 25/25 [00:01<00:00, 14.49it/s]\n",
      "Epoch 0 ended.\n",
      "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 549846.5508\n",
      "\tDiskUsage_MB/train_phase/train_stream/Task000 = 549846.5508\n",
      "\tLoss_Epoch/train_phase/train_stream/Task000 = 2.1466\n",
      "\tLoss_MB/train_phase/train_stream/Task000 = 0.6736\n",
      "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0008\n",
      "\tTime_Epoch/train_phase/train_stream/Task000 = 1.6731\n",
      "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.3575\n",
      "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.9734\n",
      "-- >> End of training phase << --\n",
      "Training completed\n",
      "Computing accuracy on the whole test set\n",
      "-- >> Start of eval phase << --\n",
      "-- Starting eval on experience 0 (Task 0) from test stream --\n",
      "100%|██████████| 20/20 [00:01<00:00, 18.24it/s]\n",
      "> Eval on experience 0 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 102.8000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 549847.1992\n",
      "\tExperienceForgetting/eval_phase/test_stream/Task000/Exp000 = 0.9844\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.8944\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0000\n",
      "-- Starting eval on experience 1 (Task 0) from test stream --\n",
      "100%|██████████| 19/19 [00:01<00:00, 18.05it/s]\n",
      "> Eval on experience 1 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 102.9170\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 549847.6338\n",
      "\tExperienceForgetting/eval_phase/test_stream/Task000/Exp001 = 0.9238\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 2.8735\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.0000\n",
      "-- Starting eval on experience 2 (Task 0) from test stream --\n",
      "100%|██████████| 21/21 [00:01<00:00, 18.22it/s]\n",
      "> Eval on experience 2 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 102.6835\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 549848.0684\n",
      "\tExperienceForgetting/eval_phase/test_stream/Task000/Exp002 = 0.8769\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 2.5142\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.0643\n",
      "-- Starting eval on experience 3 (Task 0) from test stream --\n",
      "100%|██████████| 22/22 [00:01<00:00, 18.86it/s]\n",
      "> Eval on experience 3 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 102.9035\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 549848.5029\n",
      "\tExperienceForgetting/eval_phase/test_stream/Task000/Exp003 = 0.4464\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 1.1353\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.5261\n",
      "-- Starting eval on experience 4 (Task 0) from test stream --\n",
      "100%|██████████| 21/21 [00:01<00:00, 17.83it/s]\n",
      "> Eval on experience 4 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 102.3160\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 549856.8809\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 0.5549\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.9607\n",
      "-- >> End of eval phase << --\n",
      "\tConfusionMatrix_Stream/eval_phase/test_stream = \n",
      "tensor([[ 956,    5,    0,    0,    0,    0,    0,   19,    0,    0],\n",
      "        [   0, 1124,    0,    0,    0,    0,    0,   11,    0,    0],\n",
      "        [ 294,  417,  129,    0,    0,    0,    0,  191,    0,    1],\n",
      "        [ 390,  326,    0,    0,    0,    0,    0,  292,    0,    2],\n",
      "        [  84,   28,    0,    0,    0,    0,    0,  829,    0,   41],\n",
      "        [ 407,  120,    0,    0,    0,    0,    0,  363,    0,    2],\n",
      "        [ 561,  183,   14,    0,    0,    0,    0,  199,    0,    1],\n",
      "        [   7,   48,    0,    0,    0,    0,    0,  973,    0,    0],\n",
      "        [ 189,  228,    0,    0,    0,    0,    0,  556,    0,    1],\n",
      "        [  45,   11,    0,    0,    0,    0,    0,  949,    0,    4]])\n",
      "\tDiskUsage_Stream/eval_phase/test_stream/Task000 = 549856.8809\n",
      "\tLoss_Stream/eval_phase/test_stream/Task000 = 1.9673\n",
      "\tStreamForgetting/eval_phase/test_stream = 0.8079\n",
      "\tTop1_Acc_Stream/eval_phase/test_stream/Task000 = 0.3186\n"
     ]
    }
   ],
   "source": [
    "from torch.optim import SGD\n",
    "from torch.nn import CrossEntropyLoss\n",
    "from avalanche.benchmarks.classic import SplitMNIST\n",
    "from avalanche.evaluation.metrics import forgetting_metrics, accuracy_metrics, \\\n",
    "    loss_metrics, timing_metrics, cpu_usage_metrics, confusion_matrix_metrics, disk_usage_metrics\n",
    "from avalanche.models import SimpleMLP\n",
    "from avalanche.logging import InteractiveLogger, TextLogger, TensorboardLogger\n",
    "from avalanche.training.plugins import EvaluationPlugin\n",
    "from avalanche.training.supervised import Naive\n",
    "\n",
    "scenario = SplitMNIST(n_experiences=5)\n",
    "\n",
    "# MODEL CREATION\n",
    "model = SimpleMLP(num_classes=scenario.n_classes)\n",
    "\n",
    "# DEFINE THE EVALUATION PLUGIN and LOGGERS\n",
    "# The evaluation plugin manages the metrics computation.\n",
    "# It takes as argument a list of metrics, collectes their results and returns\n",
    "# them to the strategy it is attached to.\n",
    "\n",
    "# log to Tensorboard\n",
    "tb_logger = TensorboardLogger()\n",
    "\n",
    "# log to text file\n",
    "text_logger = TextLogger(open('log.txt', 'a'))\n",
    "\n",
    "# print to stdout\n",
    "interactive_logger = InteractiveLogger()\n",
    "\n",
    "eval_plugin = EvaluationPlugin(\n",
    "    accuracy_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n",
    "    loss_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n",
    "    timing_metrics(epoch=True, epoch_running=True),\n",
    "    forgetting_metrics(experience=True, stream=True),\n",
    "    cpu_usage_metrics(experience=True),\n",
    "    confusion_matrix_metrics(num_classes=scenario.n_classes, save_image=False,\n",
    "                             stream=True),\n",
    "    disk_usage_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n",
    "    loggers=[interactive_logger, text_logger, tb_logger]\n",
    ")\n",
    "\n",
    "# CREATE THE STRATEGY INSTANCE (NAIVE)\n",
    "cl_strategy = Naive(\n",
    "    model, SGD(model.parameters(), lr=0.001, momentum=0.9),\n",
    "    CrossEntropyLoss(), train_mb_size=500, train_epochs=1, eval_mb_size=100,\n",
    "    evaluator=eval_plugin)\n",
    "\n",
    "# TRAINING LOOP\n",
    "print('Starting experiment...')\n",
    "results = []\n",
    "for experience in scenario.train_stream:\n",
    "    print(\"Start of experience: \", experience.current_experience)\n",
    "    print(\"Current Classes: \", experience.classes_in_this_experience)\n",
    "\n",
    "    # train returns a dictionary which contains all the metric values\n",
    "    res = cl_strategy.train(experience)\n",
    "    print('Training completed')\n",
    "\n",
    "    print('Computing accuracy on the whole test set')\n",
    "    # test also returns a dictionary which contains all the metric values\n",
    "    results.append(cl_strategy.eval(scenario.test_stream))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "\n",
    "## 🤝 Run it on Google Colab\n",
    "\n",
    "You can run _this chapter_ and play with it on Google Colaboratory: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ContinualAI/avalanche/blob/master/notebooks/from-zero-to-hero-tutorial/07_putting-all-together.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
